Efficient Estimators for Generalized Additive Models

Generalized additive models are a powerful generalization of linear and logistic regression models. In this paper we show that a natural regression graph learning algorithm efficiently learns generalized additive models. Efficiency is proven in two senses: the estimator’s future prediction accuracy approaches optimality at rate inverse polynomial in the size of the training data, and its runtime is polynomial in the size of the training data. Furthermore, the guarantees are nearly linear in terms of the dimensionality (number of regressors) of the problem, and hence the algorithm does not suffer from the “curse of dimensionality.” The algorithm is a simple generalization of Mansour and McAllester’s classification algorithm that generates decision graphs, i.e., decision trees with merges. Our analysis is also viewed as defining a natural extension of the original classification boosting theorems (Schapire, 1990) to the regression setting. Loosely speaking, we define a weak correlator to be a real-valued predictor that has a correlation coefficient with the target function that is bounded from zero. We show how to efficiently boost weak correlators to get predictions with correlation arbitrarily close to 1 (error arbitrarily close to 0). Our boosting analysis is a natural extension of the classification boosting analysis of Kearns and Mansour (1999) and Mansour and McAllester (2002).